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Background 

• Goal: Provide data on the effect of various Detect and Avoid 
(DAA) display features with respect to pilot performance of 
the self-separation function in order to determine the 
minimum information requirements for DAA displays 

1. What is the pilot contribution to the self-separation timeline in 
terms of expected response time to detect, determine and execute 
maneuver in response to a potential loss of well clear? 

2. What configuration of display elements meets a minimum 
acceptable level of performance? What, if any, level of pilot 
maneuver guidance is required to support this performance? 




Background 


• Display Types: 

- Informative: Provides essential information of a hazard that 
the remote pilot may use to develop and execute an 
avoidance maneuver. No maneuver guidance or decision 
aiding is provided to the piiot. 

— Suggestive: Provides a range of potentiai resoiution 
maneuvers to avoid a hazard with manuai execution. An 
algorithm provides the pilot with maneuver decision aiding 
regarding advantageous or disadvantageous maneuvers. 

- Directive: Provides specific recommended resoiution 
guidance to avoid a hazard with manuai or automated 
execution. An algorithm provides the pilot with specific 
maneuver guidance on when and how to perform the 


maneuver. 


Background 

• Approach: Conduct a series of iterative human in the loop 
experiments, in a representative simulation environment, with 
different display configuration to objectively measure pilot 
performance on maintaining well clear 

- Key metrics: pilot response time, losses of well clear, severity of losses of 
well clear 

— Three simulations have been conducted: PT4, iHITL, PT5 

• Displays are modified/improved/changed based on data/observations 

• Displays are carried through to new HITLs to create anchors or linkages to 
previous data for comparison 

• New displays are developed for test 

• Test/simulation environment/protocols also updated and improved between 
HITLs 



Simulation Environment 


• Emulation of representative environment: 

— UAS Ground Control Station (GCS) with DAA Display 

- DAA system components: 

• Surveillance 

• Threat detection and alerting 

• Suggestive and directive guidance 

- Air Traffic Control 

- Simulated Manned Traffic 

• Integrated via NASA's Live, Virtual, Constructive (LVC) 
architecture 



Simulation Environment: 
Ground Control Station (GCS) 


The Vigilant Spirit Control Station 
(VSCS) developed by the Air Force 
Research Laboratory (AFRL) 

Main Features: 

- Robust, flexible interface 

- Realistic control and navigation displays 

- System status and health monitoring 

- STANAG 4586 Compliant 

- Multi-UAS control with VSCS has been 
tested in simulation and flight by AFRL 

Current LIAS in the NAS version 
modifications/additions: 

- Single pilot - single UAS control 

- NAS-compatible database (low- and 
high- altitude charts with navigational 
aids/"fixes") 

- Integrated traffic display 7 





Simulation Environment: 

DAA System 

• The Java Architecture for DAA Modeling and 
Extensibility (JADEM) was developed by the DAS in the 
NAS project at NASA Ames Research Center 

• Main Functions: 

- Emulate surveillance parameters for various sensor types 

• e.g., ADS-B, active radar, TCAS, etc. 

- Receive state information from simulated traffic (MACS) 

• Determine which aircraft to show on traffic display(s) based on 
surveillance parameters 

- Receive trajectory information from UAS ownship (VSCS) 

- Queries all intruders for potential conflicts with ownship 

- Assigns intruders alert levels based on given thresholds 

- Host self-separation and collision avoidance algorithms 
which can provide conflict resolution guidance 




Simulation Environment: 
Draft MOPS Alerting Structure 



Name 


Self Separation 
Warning Alert 


Pilot Action 


Buffered Well 
Clear Criteria 


Time to Loss of Well 
Clear 


Aural Alert 
Verbiage 


Immediate action required 

Notify ATC as soon as practicable after 
taking action 


DMOD = 0.75 nmi 
HMD = 0.75 nmi 
ZTHR = 450ft 
modTau = 35 sec 


25 sec 

(TCPA approximate: 
60 sec) 


"Traffic, 

Maneuver 

Now" 


• On current course, corrective action 
Corrective Self required 

Separation Alert • Coordinate with ATC to determine an 
appropriate maneuver 


DMOD = 0.75 nmi 
HMD =0.75 nmi 
ZTHR = 450ft 
modTau = 35 sec 


75 sec 

(TCPA approximate: 
110 sec) 


"Traffic, 

Separate" 


Preventive Self 
Separation Alert 


On current course, corrective action 
shouid not be required 
Monitor for intruder course changes 
Talk with ATC if desired 


DMOD = 0.75 nmi 
HMD = 1.0 nmi 
ZTHR = 700ft 
modTau = 35 sec 


75 sec 

(TCPA approximate: 
110 sec) 


"Traffic, 

Monitor" 


Self Separation 
Proximate Alert 


DMOD = 0.75 nmi 

Monitor target for potential increase in HMD = 1.5 nmi 
threat level ZTHR = 1200 ft 

modTau = 35s 


85 sec 

(TCPA approximate: 
120 sec) 


N/A 


No action expected 


Within surveillance 
field of regard 


N/A 


None (Target) 










Simulation Environment: 

Multi Aircraft Control Station (MACS) 



Air Traffic Control Station 


• The Multi Aircraft Control Station 
(MACS) developed by the Airspace 
Operations Laboratory (AOL) at 
NASA Ames Research Center 

• Provides emulation of ground- and 
air- side Air Traffic Control (ATC) 
operations 

- Air Traffic Controller work stations 

- Simulated traffic generator 

- Psuedo pilot work stations 

- IFR and VFR simulated traffic 

- Traffic scenarios in Oakland Center 
(ZOA 40/41) airspace based on 
current day traffic patterns 
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Pseudo Pilot Station 






Simulation Environment: 

Multi Aircraft Control Station (MACS) 



Oakland Center ZOA 40/41 

- Class A&E 

- Current day IFR and VFR traffic 
flows 

UAS mission scenario derived 
from FAA CONORS scenarios 
(combination of "Loiter for 
Surveillance" and "Grid Pattern") 



Simulation Environment: 
LVC Architecture 


SaaProc Input: 

• Traffic 

• Ownship 

SaaProc Output: 

• Intruders 

• Saa Threat Alerts and 
Resolutions 


SaaProc/JADEM 

(sensor model) 
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Ownship 


Traffic 


Intruders 



/V 


Stratway Input: 

• Intruders 

• Ownship 

Stratway Output: 

• Stratway Bands Msg 


Traffic: 

Ownship: 

• Fit State, 

• Fit State, 

• Fit Plan, 

• Fit Plan, 

• Traj. Intent 

• Traj. Intent 

Intruders: Fit State 
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O 


re 


ADRS 

(ARC) 


Traffic 


Ownship 


ATC & Pseudo 
Pilot 
System 

(MACS) 


ATC & PPilots Input: 

• Ownship 

ATC & PPilots Output: 

• Traffic 









Background 

• Approach: Conduct a series of iterative human in the loop 
experiments, in a representative simulation environment, with 
different display configuration to objectively measure pilot 
performance on maintaining well clear 

- Key metrics: pilot response time, losses of well clear, severity of losses of 
well clear 

— Three simulations have been conducted: PT4, iHITL, PT5 

• Displays are modified/improved/changed based on data/observations 

• Displays are carried through to new HITLs to create anchors or linkages to 
previous data for comparison 

• New displays are developed for test 

• Test/simulation environment/protocols also updated and improved between 
HITLs 
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TOTAL RESPONSE TIME: 

Detect Intruders 
Pilots Determine Resolution 
Negotiate Clearance with ATC and uplink 
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PT4 - Experimental Design 


• Goal: Evaluate candidate Detect and Avoid (DAA) displays and 
algorithms with respect to self-separation and collision avoidance. 

- What are the appropriate alerting thresholds for self separation? 

- What are the minimum information requirements for DAA displays? 

- Is there a performance difference between integrated and standalone 
displays? 

- What advanced display features improve pilot performance on 
maintaining well clear from other traffic? 

• What advanced display features improve pilot performance on 
maintaining well clear from other traffic? 

- Experimental Design: Mixed Factorial Design 

- 2 (Display: Standalone, Integrated) 

- X 2 (Information: Basic, Advanced) 

- X 2 (Self-Separation Alerting Threshold) 


PT4 - Information Level 


• Display Infornnation Level: Basic versus Advanced 

1. Basic presents minimum information requirements only 

• Implementation identical as possible between Standalone and Integrated 
displays 

• Based on separate literature/requirements reviews by NASA and AFRL HMI 
teams 

• Vetted with FAA tech center (based on study they were running) 

• Similar to DO-317B (was a source document) 

• Alerting considered part of the min set 

2. Advanced information elements: 

• Implementation different between Standalone and Integrated displays 

• Additional alerting information (predictive CA) 

• Time to and location of predicted CPA (intruder and ownship) 

• Pilot guidance 

- Trial/vector planner (suggestive) 

- Maneuver recommendations (directive) 

• Vertical situation display (Integrated only) 


PT4- Standalone Displays 


Basic 


Advanced 






PT4- Integrated Displays 


Basic 


Advanced 











Average Time (s) 



PT4- Total Response Time Results 



■ Standalone ■ Integrated 



Basic Information Advanced 


There was a significant main effect of Information 
on Total Response Time, p < .05 

— Advanced was significantly faster (by 13.79 seconds 
on average) compared to Basic 

Pilots took an average of 37.87 seconds to 
complete their final edit in response to SS/CA 
alerts (from first alert appearance) 

— Basic = 47.77 sec 

— Advanced = 33.98 sec 


There was not a significant interaction of 
Information by Display for Total Response Time, p 
>.05 

Pilots took an average of 37.87 seconds to 
complete their final edit in response to SS/CA 
alerts (from first alert appearance) 

- Basic standalone = 38.68 sec 

- Basic Integrated = 44.86 sec 

- Advanced Standalone = 35.60 sec 

- Advanced Integrated = 32.35 sec 
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PT4 - Response Time Results 
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43 
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PT4 - Losses of Well Clear 


Proportion of Losses of Well Clear 



0.0 


Standalone 


Integrated 


Actual TV Severity 


PT4 - WCV Severity 



PT4 - Results Summary 


• Consistent advantage seen for Advanced over Basic displays 
in pilot response times 

- Overall, the Advanced displays had a faster Total Response Time 
(from initial alert appearance to the final maneuver upload) 
compared to Basic (14s faster, on average) 

• There were no significant differences between the 
Standalone and Integrated condition 

• No significant differences in number of, or severity of, losses 
of well clear, however: 

- Advanced had lower rates of LoWC than basic 

- No difference between Standalone and Integrated in rates of 
LoWC 

- Severity of well clear about the same across all displays 



iHITL- Experimental Design 


• Goals: 

1) Determine the individual contributions of the various PT4 advanced display 
features to pilots' response times and ability to maintain well clear 

2) Introduce non-cooperative intruders to examine effect of different sensor 
ranges on pilots ability to maintain well clear 

• One-Way Repeated Measures Factorial: Display Information Level (4 Level; 
Within Subjects) 

- Dl: Advanced Display with Information Only (Informative) 

- D2: Advanced Display with Information + Vector Planner (Suggestive) 

— D3: Advanced Display with Information + Auto Resolutions (Directive) 

— D4: Advanced Display with Information + Vector Planner + Auto Resolutions 
(Suggestive + Directive) 

• Roughly same as Advanced' suite in PT4 

• Embedded Variable 

- Intruder Equipage (manipulated within each scenario) 

• Transponder-equipped (detected via UAS's ADS-B) 

• No Transponder (detected via UAS's on-board RADAR) 



iHITL- Display Conditions 








iHITL- Total Response Time Results 


Predictive SS Alerts 



Predictive SS = encounters that are predicted to 
lose well clear at any point during the encounter 

There was a near significant effect of Display on 
Total Response Time for Predictive SS alerts, p = 
.056 

Pilots took an average of 16.22 seconds to 
complete their final edit in response to Predictive 
SS alerts (from first alert appearance) 


IHITL and PT4 Display Comparison (All Encounters) 
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PT4 - Response Time Results 
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iHITL- Response Time Results 
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iHITL - Losses of Well Clear 


Proportion of Losses of Well Clear 
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iHITL-LoWC Severity 



iHITL- Results Summary 


• Total Response Time: 

- No significant differences between displays 

- Trend shows Directive Only and Suggestive + Directive as faster than 
Information Only and Suggestive Only 

• Well Clear Metrics: 

- No significant differences between displays 

- Information and Suggestive Only (D1 and D2) display conditions had 
2.5X as many LoWCs than the Suggestive + Directive combined (D4) 

— Severity data shows evidence of trends toward performance benefits 
with Suggestive + Directive compared to other three displays 


PT5 - Overview 


• Goal: Continue evaluation of candidate Detect and Avoid (DAA) 
displays and algorithms with respect to self-separation and collision 
avoidance to inform SC-228 DAA Minimum Operational Performance 
Standards 

• Method: 

- Build upon results of previous hiti simulations results and lessons 
learned to identify minimum DAA display and guidance requirements 
for draft SC228 MOPS 

• PT4: Advanced better than Basic (but issues; well clear & display training, pop- 
ups) 

• iHITL: No significant differences between Advanced information features from 
PT4, but trends favoring combined Suggestive + Directive (D4) guidance 

• Maneuver Study (AFRL): Banding display showed faster response time 
compared to informative and directive displays; banding and advanced 
informative had least losses of well clear (neither results statistically significant) 


PT5 - Experimental Design 


• Mixed Factorial Design 

— Display Configuration (Within-Subjects Independent Variable): 

• Configuration 1: Minimum Information Set (No Guidance) 

• Configuration 2: Stratway+ No Fly Bands 

• Configuration 3: JADEM Omni Bands 

• Configuration 4: JADEM Vector Planning Tools 

- Sensor Performance (Between-Subjects Independent Variable) 

• Level 1: Perfect Surveillance Data 

• Level 2: Imperfect Surveillance Data 

• Embedded Variable 

— Intruder Equipage (manipulated within each scenario) 

• Transponder-equipped (detected via UAS's ADS-B) 

• No Transponder (detected via UAS's on-board RADAR) 
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PT5 - Display Conditions 
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PT5- Total Response Time Results 
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Info Only Stratway+ Omni Bands Vector Planner 
Configuration 


• Pilots responded, on average, 10s faster to SS Warning Alerts than they did to 
Corrective SS Alerts 

- Pilots exhibited less variability between displays when responding to SS Warning Alerts than to 
Corrective SS Alerts 

• Range for SS Warning Alerts: 11s - 15s 

• Range for Corrective SS Alerts: 19s - 30s 

- Variability due to coordination with ATC - adds ~ 10 secs to total response time 
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• Pilots responded, on average, 4.5s faster to non-cooperative traffic than they did to 
cooperative traffic, which was a significant difference (p=.008) 

- There was also less variability in pilots' responses to Non-Cooperative encounters 

• Sensor model was not found to have any effect on pilot's Total RTs 
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PT5 -Losses of Well Clear 
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Info Stratway+ Omni Bands Vector Planner 

Configuration 


• Info Only ( 19 . 8 %) was roughly /ot/r times as likely as Stratway+ ( 6 . 5 %) and Omni Bands 
( 4 . 2 %) to result in LoWC, a significant difference (p<.05) 

- Info Only was roughly two times as likely as Vector Planner (10.3%) to lead to LoWC, which 
approached significance (p=.086) 

• No significant differences seen between the three guidance displays in terms of LoWC 
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PT5 - LoWC Severity 


• All Displays: 

- Actual Separation / Separation Threshold 

- Less than 1 = spatial separation was NOT maintained 



0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 2 


Severity Index 


No Separation 
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PT5 - Results Summary 


• Suggestive guidance in the form of banding resulted in safer 
and more timely maneuvers away from conflicts 

- Fewer overall number of LoWC for both banding displays 

- Less severe LoWC for both banding displays 

- Shorter Total RTs for both banding displays 

- Pilots self-report as preferring the banding displays 

• Results support decision for suggestive guidance as a minimum 
information requirement for DAA displays 

• Results indicate that pilots can respond to a DAA Warning alert 
(no ATC coordination required) in ~ 15 seconds 

• Results indicate that pilots can respond to a DAA Corrective 
alert (ATC coordination is required) in ~ 25 seconds 

• ATC coordination adds approximately 10 seconds to DAA 
timeline 



Suggestive Guidance Display - Example 






PT4- Total Response Times 
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iHITL- Total Response Times 




4 > 

Aircraft 

Maneuver Time 


D1 (21s) 


54 sec 




D2 (20s) 


55 sec 

D3 (16s) 


59 sec 

D4 (16s) 


59 sec 


Well Clear 
Threshold 


47 



PT5 - Total Response Times 




♦ 


» 

Aircraft well Clear 

Maneuver Time Threshold 





48 


Self-Separation Timeline 
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Questions? 


Backup Slides 



Sensor Parameters 


> Sensor Ranges 

• Simulated cooperative sensor: ADS-R/TCAS-like ranges 

• Lateral Range: 15 nm 

• Vertical Range: +/- 5000 ft 

• Simulated non-cooperative sensor: based on state-of-the-art 
airborne RADAR 

• Lateral Range: 8 nm 

• Azimuth: +/- HO degrees surveillance Range 

• Ele 


Ver1 
Field of 

(Elevation Angle) 



Horizontal 
Field of Regard 
(Azimuth Angle) 


Parameters for Noisy Cooperative Sensor 


Noisy Cooperative Sensor ("Transponder") 


Parameter 

Value 

Unit 

Field Of Regard 

Range 

15 

nmi 

Azimuth 

360 

deg 

Elevation 

+/-90 

deg 

Accuracy 

Range Error Mean 

0 

nmi 

Range Error Std. Dev 

0 

nmi 

Range Moving Avg. Window Size 

1 

measures 

Azimuth Error Mean 

0 

deg 

Azimuth Error Std. Dev 

2 

deg 

Azimuth Moving Avg. Window Size 

3 

measures 

Altitude Quantization 

100 

feet 

Altitude Moving Avg. Window 

6 

measure 


Yellow denotes the noise nnodel variables that will used for PT5. 
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Parameters for Noisy Non-Cooperative Sensor 


Noisy Non-Cooperative Sensor ("Airborne Radar") 



Parameter 

Value 

Unit 

Field Of Regard 

Range 

6 

nmi 

Azimuth 

+/-110 

deg 

Elevation 

+/-20 

deg 

Accuracy 

Range Error Mean 

0.008 

nmi 

Range Error Std. Dev. 

0.001 

nmi 

Range Moving Avg. Window Size 

1 

measures 

Azimuth Error Mean 

0 

deg 

Azimuth Error Std. Dev. 

2 

deg 

Azimuth Moving Avg. Window Size 

3 

measures 

Elevation Error Mean 

1 

deg 

Elevation Error Std. Dev. 

1 

deg 

Elevation Moving Avg. Window Size 

6 

measure 


Yellow denotes the noise model variables that will used for PT5. 


Parameters for "Perfect" Cooperative Sensor 


"Perfect" Cooperative Sensor ("ADS-B") 


Parameter 

Value 

Unit 

Field Of Regard 

Range 

15 

nmi 

Azimuth 

360 

deg 

Elevation 

+/-90 

deg 

Accuracy 

Latitude Error 

0 

deg 

Latitude Error Std. Dev. 

0 

deg 

Latitude Moving Avg. Window Size 

1 

measures 

Longitude Error 

0 

deg 

Longitude Error Std. Dev. 

0 

deg 

Longitude Moving Avg. Window Size 

1 

measures 

Altitude Error 

0 

deg 

Altitude Error Std. Dev. 

0 

deg 

Altitude Moving Avg. Window Size 

1 

measure 


Parameters for "Perfect" Non-Cooperative Sensor 


"Perfect" Non-Cooperative Sensor ("Perfect Airborne Radar") 



Parameter 

Value 

Unit 

Field Of Regard 

Range 

6 

nmi 

Azimuth 

+/-110 

deg 

Elevation 

+/-20 

deg 

Accuracy 

Range Error Mean 

0 

nmi 

Range Error Std. Dev. 

0 

nmi 

Range Moving Avg. Window Size 

1 

measures 

Azimuth Error Mean 

0 

deg 

Azimuth Error Std. Dev. 

0 

deg 

Azimuth Moving Avg. Window Size 

1 

measures 

Elevation Error Mean 

0 

deg 

Elevation Error Std. Dev. 

0 

deg 

Elevation Moving Avg. Window Size 

1 

measure 


